As part of our LLM prototyping work, Consumer Reports (CR) is experimenting with different ways to query our data sources using plain english requests. Our goal is to translate a colloquial query into a database query that returns the correct results from CR’s data sources. This is an interesting case study for LLM retrieval since there are objectively right and wrong answers to questions. For example, there is one right answer to “Which lightweight snowblower had the highest overall CR rating?”
As part of this experimentation, we’ve tried using python interpreters to read data directly from CR ratings & reviews via pandas dataframes. We wanted to understand whether there were any security implications we should consider when using this technology. To help us we reached out to an expert from Include Security who specializes in LLM security.
IncludeSec’s findings reveal how a thorny security problem can appear to be resolved, without actually ever being addressed – not in bad faith, but simply because automated tools and the touch-and-go nature of open source development lead the problem to fall through the cracks. Read the beat by beat of IncludeSec’s investigation in their own words (in italics below).
Be warned that we plan to get really technical and in the weeds here (it’s an epic tale for security engineers who love the nitty gritty), but plan to come back to the surface and share universal takeaways for responsible development at the end.
IncludeSec: There are certain popular LLM components known to be problematic and difficult to secure among security experts. Components that receive a user query/prompt and translate that into code, SQL or other dynamic formats for evaluation are top of that list. These components are problematic because an attacker can influence the generation of executed code, SQL, or other formats of syntax via their input prompt.
While undertaking a security assessment of an LLM-based application for Consumer Reports, we learned of a component CR was using which dynamically generated dataframe code for execution at runtime. The fact that it was executing dynamically generated dataframe code made our spidey senses tingle – but it was an architecture review and we (at the time) didn’t have source code. As such we first tried doing a quick search of popular Common Vulnerabilities Exposures (CVE) databases, and nothing came up on the first page of results. This was intriguing!
During the threat modeling portion of the assessment we had the opportunity to review the source code and decided to do some digging. The component in question was a part of LangChain called the Panda Dataframe Agent. There was a CVE (NVD – CVE-2023-39659 nist.gov) associated with the component and the recommendation was to upgrade to langchain-ai v.0.0.325 or higher.
Though there was a patch referenced, we wanted to investigate how it would even be possible to fix this problem. So we dug deeper into the source code of the Panda Dataframe Agent code and found this:
#From PythonAstREPLTool langchain_experimental.tools.python.tool — 🦜🔗 LangChain 0.1.16
The code is a little confusing, but it basically executes ( exec() ) all the code except for the last line (as system commands) generated from the LLM and then evaluates ( eval() ) the last line of code (as python). We also notice the sanitize function, which should be doing something to reduce risk; however we found that it only removes spaces and “python” from the beginning of the code, as well as backtick marks around the code:
We dug further because the national vulnerabilities database (NVD) had suggested that there was a patch that fixed this. It was hard to follow because there were a number of RCE (Remote Code Execution) problems in different LangChain components (LLMMathChain, SQLDatabaseChain, Panda Dataframe Agent, PythonAgent, SparkAgent) that looked to be tied to this one CVE-2023-39659.
Doing historical analysis of the code base we identified a number of attempts to fix the issue. Here is one of the earlier attempts:
From Patch LLMMathChain exec vulnerability by zachschillaci27 · Pull Request #1119 · langchain-ai/langchain · GitHub
This fix was almost as bad as the initial problem. The developer noticed that some of the attack payloads in the original bug ticket were stopped by the eval() method above, however, attackers can still run arbitrary system commands with python code by shelling out to the system and executing system commands. For example:
Luckily other developers chimed in that the solution would not be a complete fix. Then the developers started discussing how to fix the problem by using Restricted Python, running everything in a scaled down container, using wasm_exec() instead of exec()… but all the discussions concluded with variations of the following:
From RFC: Use wasm-exec package to sandbox code execution by the Python REPL tool · Issue #5388 · langchain-ai/langchain · GitHub
From Issue: security concerns with `exec()` via multiple agents and Shell tool · Issue #5294 · langchain-ai/langchain · GitHub
From Security concerns · Issue #1026 · langchain-ai/langchain · GitHub
Reviewing the threads associated with the three dosubot replies above, it’s clear that there was a flurry of activity as different parties tried to implement solutions. Eventually things quiet down, then dosubot comes around and kills the three distinct efforts without anything being resolved or any discussion of what could be done to incorporate the potential fixes into LangChain. This is a major concern for security fixes, many open source projects have similar bots which prematurely terminate efforts to fix tough problems for the sake of issue tracker organization.
Finally, the resolution revealed itself:
Deprecate PythonRepl tools and Pandas/Xorbits/Spark DataFrame/Python/CSV agents (CVE-2023-39659) by eyurtsev · Pull Request #12427 · langchain-ai/langchain · GitHub
SQLDatabaseChain has SQL injection issue · Issue #5923 · langchain-ai/langchain · GitHub
The commits above indicated that developers were deprecating the problematic code, but what actually happened was that the code was moved to an experimental directory within the LangChain github repository. The commit messaging and the code change contradict each other, which may cause readers to think the problem is fixed when it in fact is fully exploitable!
It wasn’t just the National Vulnerabilities Database that was caught off guard by this. Synk’s Vulnerability Database issued similar guidance and appears to also be incorrect:
But you can still run the latest version of LangChain code with the proof of concept example code on the Synk website to see that it is still exploitable:
Arbitrary Code Execution in langchain | CVE-2023-39659 | Snyk
Upon discovering this, IncludeSec had several brainstorming sessions with CR to work through the problem and craft a remediation solution fit for CR’s use case.
Dan from CR: Given the clear security issues that IncludeSec worked with us to identify in a key dependency, CR reassessed our need for a python code interpreter and have begun experimenting with alternative approaches.
The engineering team at CR had two major takeaways from this exercise. First, it is important to engage security experts early in the LLM development lifecycle. Involving security in LLM architecture review gave us a premonition of where potential vulnerabilities could exist in our system, and allowed us ample time to investigate code-level issues and brainstorm alternate approaches.
Second, some security issues that appear in vulnerability databases and present as resolved may not actually be patched. LLM tooling is developing at a meteoric pace, and the hardest problems can sometimes get lost as developers prefer to direct their attention towards new feature development and quick wins. CR plans to do our homework on future vulnerabilities we identify to double-check that patches really do address their underlying security issues.
If you’re a security expert interested in hardening LLM applications, we’re keen to hear from you. You can reach out at innovationlab@cr.consumer.org.
Thanks to Abraham Kang for the deep partnership on this project, and to Erik Cabetas and Kensington Moore from Include Security for their support.
